1. Field of the Invention
The invention relates to neural networks, and more particularly, to customizing the same both architecturally and in terms of synaptic weights, and to a technique for fabricating synaptic resistors usable in such a device.
2. Description of Related Art
In a typical neural network, a plurality of input neurons (which may simply be isolation buffers) receive input excitation signals. The outputs of the input neurons are coupled to selected ones of the inputs of a plurality of second level neurons via synaptic interconnection circuits or synapses. Different ones of the input signals combine with different weights at the inputs of the second level neurons, and the particular weight to be accorded each signal is governed by a setting in the synaptic interconnection circuit which couples that signal to that neuron. The outputs of the second level neurons may themselves further be coupled through another set of synaptic interconnection circuits, with selected weights, to inputs of selected ones of a plurality of third level neurons. In this case the second level neurons are often referred to as "hidden" neurons. The outputs of the third level neurons may be provided as the outputs of the overall network, in which case these neurons may be referred to as "output neurons", or they may be coupled to yet additional layers of neurons by additional selectively weighted synaptic interconnection circuits. In addition, in some neural network architectures, outputs of some of the neurons are fed back to the inputs of a prior level.
Neural networks may have a fixed interconnection pattern and fixed synaptic weights, or the synaptic weights may be made variable. If the synaptic weights are made variable, then the network may be given the capacity to "learn". For example, Tsividis U.S. Pat. No. 4,903,226 describes a network in which the neurons are all made up of operational amplifiers and the synaptic interconnection of circuits are merely MOS transistors whose series resistance is controlled by the charge on a capacitor connected to the gate. Currents from the outputs of several preselected neurons pass through their respective synaptic interconnection resistances and are combined at the input of the destination neuron; thus the resistance of each synaptic interconnection circuit defines the weight which an input signal is accorded at the input of the destination neuron. Once these resistance values are known for a particular application, a production version of the network may be mass produced using fixed resistors in place of the variable synaptic interconnection resistances.
One problem with variable resistance synaptic interconnection circuits is that they are generally highly complex and not suitable for integrating the huge number of synapses which are probably required to mimic an activity of the human brain with any degree of success. The human cerebral cortex, for example, contains approximately 10.sup.14 synapses, a factor which may well be responsible for the powerful cognitive capabilities exhibited by human beings. Fixed resistor synaptic arrays may be a more promising approach toward such high densities, but each different application for a neural network requires its own set of synaptic interconnection weights. Most applications do not support the large production volumes which would be required to make such a neural network integrated circuit economical.
Accordingly, it is desirable to provide a customizable neural network chip which can be produced in large volume and subsequently be customized for a particular application. Such a device would be useful both for manufacturing production networks for low volume applications, and also for prototyping neural networks for applications which will eventually support the production volumes of a custom chip.
Tam U.S. Pat. No. 4,961,002 describes a synaptic array in which each synapse cell connecting an input and an output line is made using a dual-gate transistor. The transistor has a floating gate member for storing electrical charge, a pair of control gates coupled to a pair of input lines, and a drain coupled to an output summing line. The floating gate of the transistor is used for storing a charge which corresponds to the strength or weight of the neural connection. Thus, when a binary signal is applied to the cell from an input line, it is multiplied in an analog manner by the charge stored on the floating gate to determine the current to be coupled to the output summing line. The Tam device, therefore, essentially contains a synaptic interconnection circuit whose weight is programmable as an EEPROM cell.
In practice, however, the input-output relationship is not linear. It is therefore difficult to perform accurate computer simulations of how the chip would respond when various synaptic weights are programmed. Additionally, since the Tam device probably must be made using an 18-mask EEPROM process, it is likely that yields are low and the cost is high.
Other neural network chips have also been attempted using a transistor as the synaptic coupling element, biased to operate in the linear (subthreshold) region of its characteristic. These devices are difficult to manufacture and the useful signal range is small.
Ordinary integrated circuit resistors have been used to interconnect two points with a programmable resistance, although not in the context of a neural network so far as applicant is aware. For example, various forms of post-fabrication resistor trimming have been devised to adjust resistance values, including laser trimming or cutting, "Zener-zapping", and metal link cutting and blowing. Laser trimming involves the use of a laser to alter the shape of a resistor region and thereby bring its resistance to the desired value. Serious problems may arise from aging and annealing effects resulting from this technique, however. The "partially zapped" material along the edge of the cut trim path often has different properties from the undisturbed material, and its resistance may age at a different rate than the body of the resistor. This can give rise to a situation where a resistor which is initially trimmed to a particular ratio with other resistors on the chip, gradually drifts out of specification during usage.
To avoid aging problems, it is known to use a trimming geometry in which resistive links are either totally cut or left undisturbed. The infinite resistance of a cut resistor is unaffected by aging. Known techniques which make use of this property include a set of resistive links which are connected in a parallel geometry. However, if N-links are used, the resolution of the trim is only 1/N. Trim resolutions can be increased by using binary-weighted resistor links, but in such geometries, the ratios of resistance values have in the past been subject to undesirable variations due to process variations during fabrication and temperature variations during use. Additionally, it is undesirable to program resistances by laser cutting of the resistor material itself, since this limits the choice of resistor materials. As will be seen, choice of an optimum resistor material can greatly improve the packing density of a synaptic array.
Still another known approach for adjusting the resistance value of the resistors on integrated circuits is one in which the resistive links are shorted by metal. The metal shorts are blown open with a current pulse or a laser beam. Again, binary weighting offers advantages.
In Shier U.S. Pat. No. 4,782,320, incorporated herein by reference, there is described (outside the context of neural networks) a technique in which two terminal leads are coupled with a resistor network in which a plurality of N-sided meshes are each formed by N resistor elements linked at network nodes. Some of the resistor elements can be cut by a laser applied to the resistive film itself to thereby program a desired overall resistance value coupling the two terminal leads. In Shier's technique, however, the resistor itself is cut by the laser, thereby limiting the types of resistor materials that can be used.
With respect to integrated resistor technology, it is desirable that resistors used in a synaptic interconnection circuit be both as small as possible and have as high a resistivity as possible. Polycrystalline resistors used in conventional silicon processing have far too low a resistivity, and would therefore require extensive meandering over a large area to form a resistor having a high enough resistance. Polycrystalline silicon resistors are also very difficult to manufacture uniformly since the grain boundaries of the polycrystalline material, which play an important role in conductivity, are difficult to control in tile fabrication process.
Accordingly, it is an object of the present invention to provide a neural network integrated circuit chip which is configurable according to the interconnections and synaptic weights required by a large number of different applications. The neural network integrated circuit should also be able to support a very large number of synaptic interconnections with uniform and linear high resistance programmable weights, all with high yield.